Systems and methods of direct pointing detection for interaction with a digital device

ABSTRACT

Systems, methods, and non-transitory computer-readable media are disclosed. For example, a touch-free gesture recognition system is disclosed that includes at least one processor. The processor may be configured to enable presentation of first display information to a user to prompt a first touch-free gesture at at least a first location on a display. The processor may also be configured to receive first gesture information from at least one image sensor corresponding to a first gesturing location on the display correlated to a first touch-free gesture by the user, wherein the first gesturing location differs from a location of the first display in formation at least in part as a result of one eye of the user being dominant over another eye of the user. In addition, the processor may be configured to determine a first offset associated with the location of the first display information and the first gesturing location. Further, the processor may be configured to enable presentation of second information to prompt the user to make a subsequent touch-free gesture at at least a second location on the display. Additionally, the processor may be configured to receive subsequent gesture into from at least one image sensor corresponding to a subsequent touch Ire gesture by the user. Also, the processor may be configured to use the first offset to determine a location on the display affected by the subsequent touch-free gesture.

RELATED APPLICATIONS

This applications claims priority to U.S. provisional application No.61/840,447, filed Jun. 27, 2013, which is incorporated herein byreference in its entirety.

TECHNICAL FIELD

The present disclosure relates to the field of touch-free gesturedetection and, more particularly, devices and computer-readable mediafor using a sensor to detect touch-free gestures.

BACKGROUND

Permitting a user to interact with a device or an application running ona device is useful in many different settings. For example, keyboards,mice, and joysticks are often included with electronic systems to enablea user to input data, manipulate data, and cause a processor of thesystem to execute a variety of other actions, increasingly, however,touch-based input devices, such as keyboards, mice, and joysticks, arebeing replaced by, or supplemented with devices that permit touch-freeuser interaction. For example, a system may include an image sensor tocapture images of a user, including, for example, a user's hand and/orfingers. A processor may be configured to receive such images andinitiate actions based on touch-free gestures performed by the user. Itis desirable to improve upon techniques for detecting and inputtingtouch-free gestures.

SUMMARY

In one disclosed embodiment, a touch-free gesture recognition system isdisclosed. The touch-free gesture recognition system may include atleast one processor. The at least one processor may be configured toenable presentation of first display information to a user to prompt afirst touch-free gesture at at least a first location on a display,receive first gesture information from at least one image sensorcorresponding to a first gesturing location on the display correlated toa first touch-free gesture by the user, wherein the first gesturinglocation differs from a location of the first display information atleast in part as a result of one eye of the user being dominant overanother eye of the user, determine a first offset associated with thelocation of the first display information and the first gesturinglocation, enable presentation of second information to prompt the userto make a subsequent touch-free gesture at at least a second location onthe display, receive subsequent gesture information from the at leastone image sensor corresponding to a subsequent touch-free gesture by theuser, and use the first offset to determine a location on the displayaffected by the subsequent touch-free gesture.

Additional aspects related to the embodiments will be set forth in partin the description which follows, and in part will be understood fromthe description, or may be learned by practice of the disclosedembodiments.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory onlyand are not restrictive of the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate several exemplary embodiments ofthe invention and together with the description, serve to explain theprinciples of the invention.

FIG. 1 illustrates an example graphical representation of a touch-freeuser interface system in accordance with some of the disclosedembodiments.

FIG. 2A illustrates an exemplary processing unit of a touch-free userinterface in accordance with some of the disclosed embodiments.

FIG. 2B illustrates an exemplary block diagram of a system in accordancewith some of the disclosed embodiments.

FIG. 3A illustrates exemplary offset determination steps that used inaccordance with methods, systems, and computer readable media associatedwith embodiments of the invention.

FIG. 3B illustrates exemplary gesture detection and identifying stepsthat may be used in accordance with methods, systems, and computerreadable media associated with embodiments of the invention.

FIG. 4 illustrates an exemplary graphical representation of a touch-freeuser interface system in accordance with some of the disclosedembodiments.

FIG. 5 illustrates exemplary steps for identifying a pointing locationthat may be used in accordance with methods, systems, and computerreadable media associated with embodiments of the invention.

FIG. 6A illustrates an exemplary graphical representation of atouch-free user interface system in accordance with some of thedisclosed embodiments.

FIG. 6B illustrates an exemplary graphical representation of atouch-free user interface system in accordance with some of thedisclosed embodiments.

FIG. 7 illustrates exemplary calibration steps that may be used inaccordance with methods, systems, and computer readable media associatedwith embodiments of the invention.

FIG. 8 illustrates an exemplary graphical representation of a touch-freeuser interface system in accordance with some of the disclosedembodiments.

FIG. 9 illustrates exemplary steps for identifying candidate planes orcandidate areas that may be used in accordance with methods, systems,and computer readable media associated with embodiments of theinvention.

FIG. 10 illustrates a Venn diagram of exemplary candidate planes inaccordance with some of the disclosed embodiments.

FIG. 11 illustrates exemplary gesture tracking steps that may be used inaccordance with methods, systems, and computer readable media associatedwith embodiments of the invention.

FIG. 12 illustrates exemplary steps for determining eye dominance thatmay be used in accordance with methods, systems, and computer readablemedia associated with embodiments of the invention.

FIG. 13 illustrates an exemplary graphical representation of atouch-free user interface system in accordance with some of thedisclosed embodiments.

FIG. 14 illustrates an exemplary graphical representation of a user'sface in accordance with some of the disclosed embodiments.

FIG. 15 illustrates an exemplary graphical representation of a user'shand in accordance with some of the disclosed embodiments.

FIG. 16A illustrates an exemplary graphical representation of a pointingvisual effect in accordance with some of the disclosed embodiments.

FIG. 16B illustrates an exemplary graphical representation of a pointingvisual effect in accordance with some of the disclosed embodiments.

FIG. 17 illustrates an exemplary graphical representation of atouch-free user interface system in accordance with some of thedisclosed embodiments.

DETAILED DESCRIPTION

Reference will now be made in detail to the exemplary embodiments, whichare illustrated in the accompanying drawings. Wherever possible, thesame reference numbers will be used throughout the drawings to refer tothe same or like parts.

FIG. 1 depicts an embodiment of a system 2 for touch-free operation of adevice 4 having a display 6. The device 4 may include, for example, amobile phone, smart-glasses, a personal computer (PC), an entertainmentdevice, a set top box, a television, a mobile game machine, a tabletcomputer, an e-reader, a portable game console, a portable computer suchas a laptop or ultrabook, a home appliance such as a kitchen appliance,a communication device, an air conditioning thermostat, a dockingstation, a game machine such as a mobile video gaming device, a digitalcamera, a watch, an entertainment device, speakers, a Smart Home device,a media player or media system, a location-based device, a picoprojector or an embedded projector, a medical device such as a medicaldisplay device, a vehicle, an in-car/in-air infotainment system, anavigation system, a wearable device, an augmented reality-enableddevice, wearable goggles, a robot, interactive digital signage, adigital kiosk, a vending machine, an automated teller machine (ATM), orany other apparatus or system that may receive data from a user oroutput data to a user.

The display 6 may include any plane, surface, or other instrumentalitycapable of causing a display of images or other visual information.Further, the display 6 may include any type of projector that projectsimages or visual information onto a plane or surface. For example, thedisplay 6 may include one or more of a television set, computer monitor,head-mounted display, broadcast reference monitor, a liquid crystaldisplay (LCD) screen, a light-emitting diode (LED) based display, anLED-backlit LCD display, a cathode ray tube (CRT) display, anelectroluminescent (ELD) display, an electronic paper/ink display, aplasma display panel, an organic light-emitting diode (OLED) display,thin-film transistor display (TFT), High-Performance Addressing display(HPA), a surface-conduction electron emitter display, a quantum dotdisplay, an interferometric modulator display, a swept-volume display, acarbon nanotube display, a variforcal mirror display, an emissive volumedisplay, a laser display, a holographic display, a light field display,a wall, a three-dimensional display, an e-ink display, and any otherelectronic device for outputting visual information. The display 6 mayinclude or be part of a touch screen. FIG. 1 depicts display 6 as partof device 4. However, in alternative embodiments, display 6 may beexternal to device 4.

The system 2 may also include (or receive information from) an imagesensor 8 positioned adjacent to the device 4 and configured to obtainimages of a three-dimensional (3-D) viewing space bounded by the brokenlines 10. The image sensor 8 may include any image acquisition deviceincluding, for example, one or more of a camera, a light sensor, aninfrared (IR) sensor, an ultrasonic sensor, a proximity sensor, a CMOSimage sensor, a shortwave infrared (SWIR) image sensor, or areflectivity sensor, a CCD image sensor, a reflectivity sensor, a depthvideo system comprising a 3-D image sensor or two or moretwo-dimensional (2-D) stereoscopic image sensors, and any other devicethat is capable of sensing visual characteristics of an environment. Byway of example, FIG. 1 depicts the image sensor 8 adjacent to the device4, but in alternative embodiments, the image sensor 8 may beincorporated into the device 4 or even located away from the device 4.

A user 16 situated in the viewing space indicated by the broken lines 10may appear in images obtained by the image sensor 8. The image sensor 8may output 2-D or 3-D monochrome, color, or IR video to a processingunit 12, which may be integral with the image sensor 8 or connected tothe image sensor 8 by a wired or wireless communication channel.

Embodiments of the invention may include at least one processor. As usedherein, the term “processor” may include an electric circuit thatperforms a logic operation on an input or inputs. For example, such aprocessor may include one or more integrated circuits, microchips,microcontrollers, microprocessors, all or part of a central processingunit (CPU), graphics processing unit (GPU), digital signal processors(DSP), field-programmable gate array (FPGA), an application-specificintegrated circuit (ASIC), or any other circuit suitable for executinginstructions or performing logic operations. The at least one processormay be coincident with or may constitute any part of a processing unit,such as, for example, processing unit 12 depicted in FIG. 2A. Theprocessing unit 12 of FIG. 2A may include, among other things, aprocessor 15 and memory 13 that may be used for storing images obtainedby the image sensor 8. The processing unit 12 and/or the processor 15may be configured to execute one or more instructions that reside in theprocessor 15 and/or the memory 13.

The memory 13 may include, for example, one or more of persistentmemory, ROM, EEPROM, EAROM, flash memory devices, magnetic disks,magneto optical disks, CD-ROM, DVD-ROM, Blu-ray media, and may containinstructions (i.e., software or firmware) and/or other data. FIG. 2Adepicts the memory 13 as part of the processing unit 12, but in otherembodiments the memory 13 may be external to the processing unit 12.

The processing unit 12 may be configured to analyze images obtained bythe image sensor 8 and track one or more predefined pointing elementsthat may be utilized by the user 16 for interacting with the display 6.A pointing element may include, for example, a fingertip 14 of a user 16situated in the viewing space of the image sensor 8. In someembodiments, the pointing element may include, for example, one or morehands of the user 16, a part of a hand, one or more fingers, one or moreparts of a finger, and one or more fingertips 14, or a hand-held stylus.Although FIG. 1 depicts the fingertip 14 as a pointing element, otherpointing elements may be similarly used and may serve the same purpose.Thus, wherever the fingertip 14 is mentioned in the present descriptionit should be considered as an example only and should be broadlyinterpreted to include other pointing elements well.

The processing unit 12 may be configured to present display information,such as an icon 20 on the display 6 towards which the user 16 may pointthe fingertip 14. The processing unit 12 may be further configured toindicate an output 18 (indicator 18) on the display 6 corresponding tothe location pointed at by the user 16. For example, as shown in FIG.17, the user 16 may intend to point at the display information (icon 20)on the display 6. In this example, the processing unit 12 may determinethat the user 16 is actually pointing at a gesturing location 19,However, the processing unit 12 may determine an offset O associatedwith the difference between the display information and the gesturinglocation 19. By using in part, information relating to the offset O, theprocessing unit 12 may send an output (e.g., indicator 18) to adestination of the display 6 reflective of the user's intent. In theexample shown in FIG. 17, the indicator 18 is displayed at the samelocation as the icon 20, despite that the user 16 is actually pointingat the gesturing location 19.

A gesturing location 19 may be any mathematical representationassociated with a location on the display 6, which is defined at somepoint by the system 2 as the location on which the user points at. Thegesturing location 19 can include a specific coronation on the display(x, y) or (x, y, z, in case of a 3-D display). The gesturing location 19can include an area or location on the display 6 (e.g., candidateplane). In addition, the gesturing location 19 can be a defined asprobability function associated with a location on the display (such asa 3-D Gaussian function). The gesturing location 19 can be associatedwith a set of addition figures, which describes the quality ofdetection, such as probability indication of how accurate the estimationof the location on the display 6 of the gesturing location 19.

In case of a smart-glass, e.g., a wearable glass that include thecapability to present to the user 16 digital information, the gesturinglocation may be defined as the location of a virtual plane, the plane onwhich the user perceived to see the digital information that ispresented by the smart-glass display.

Display information may include static images, animated images,interactive objects (such as icons), videos, and/or any visualrepresentation of information. Display information can be displayed byany method of display as described above and may include flat displays,curved displays, projectors, transparent displays, such as one used inwearable glasses, and/or displays that projects directly to or indirectly to the user's eyes or pupils.

Indication or feedback of the pointed-at icon may be provided by, forexample, one or more of a visual indication, an audio indication, atactile indication, an ultrasonic indication, and a haptic indication.Displaying a visual indication may include, for example, displaying anicon on the display 6, changing an icon on the display 6, changing acolor of an icon on the display 6, displaying an indication light,displaying highlighting, shadowing or other effect, moving an indicatoron a display 6, providing a directional vibration indication, and/orproviding an air tactile indication. The visual indicator may appear ontop (or in front of) all other images or video appearing on the display6. A visual indicator, such as icon on the display 6 selected by theuser 16, may be collinear with the user's eye 22 and the fingertip 14lying on a common viewing ray 24 (or line of sight). As used herein, andfor reasons described later in greater detail, the term “user's eye” isa short-hand phrase defining a location or area on the user's faceassociated with a line of sight. Thus, as used herein, the term “user'seye” encompasses the pupil of either eye or other eye feature, alocation of the user face between the eyes, or a location on the user'sface associated with at least one of the user's eyes, or some otheranatomical feature on the face that might be correlated to a sight line.This notion is sometimes also referred to as a “virtual eye”.

An icon is an exemplary graphical element that may be displayed on thedisplay 6 and selected by a user 16. In addition to icons, graphicalelements may also include, for example, objects displayed within adisplayed image and/or movie, text displayed on the display 6 or withina displayed file, and objects displayed within an interactive game.Throughout this description, the terms “icon” and “graphical element”are used broadly to include any displayed information.

As depicted in step 313 of FIG. 3A, embodiments of the invention mayenable presentation of first display of information to a user 16 toprompt a first touch-free gesture at at least a first location on adisplay 6. Enabling presentation may occur, for example, by transmittingdata from a processor 15 to a display device, in order to cause theinformation or a representation thereof to be presented to the user 16via the display 6. The representation may be designed to evoke or promptthe user 16 to make a touch free gesture toward the display 6, asillustrated, by way of example only, in FIG. 1. In particular, FIG. 1depicts a user 16 performing a finger pointing touch-free gesture. Thefirst location on the display 6 may generally correspond to the locationof display information. The correspondence may not be precise as theremay be differences between the user's perception (i.e., the displayedinformation)and the actual location of the displayed information (i.e.,the gesturing location), as described later in greater detail. The firstinformation may include, for example, graphical elements, icons, orother information, as described in more detail below.

For example, the processing unit 12 may also be configured to display onthe display 6 one or more selectable graphical elements or icons. FIG. 1illustrates four exemplary icons 20 a, 20 b, 20 c, and 20 d. However, insome embodiments, any number of icons may be displayed on the display 6at any time. Any one of the icons 20 a-20 d may be selected by the user16 by pointing the fingertip 14 at a particular icon the user 16 intendsto select.

As depicted in step 315 of FIG. 3A, embodiments of the invention mayreceive first gesture information corresponding to a first gesturinglocation on the display 6 correlated to and reflective of a firsttouch-free gesture by a user 16. First gesture information cancorrespond to any motion or pose, as described later in greater detail.The Gesture information may include, for example, numerical or graphicaldata corresponding to a detected hand-gesture or pointing element. Thenumerical or graphical data may be received by at least one processor15, and the processor 15 may correlate it to a known display location.

The processing unit 12 may be configured to identify a selection of anicon by the user 16 based on a predefined criterion. For example, whenthe user16 points at the icon for a predefined period of time apredefined number of seconds), the processing unit 12 may select theicon. In addition, the processing unit 12 may be configured to identifya selection of an icon by the user 16 when the user 16 performs apredefined motion towards the selected icon. For example, the predefinedmotion may include a tapping motion, wherein a movement of the fingertip14 towards the selected icon is followed by a movement of the fingertip14 away from the selected icon. The system 2 may be configured togenerate a visual and/or audio feedback when the user 16 points to andselects an icon. If, for example, a single icon is located on a display6 and a user gesture is detected, the system 2 may be configured toassume that the user 16 has gestured toward the only displayed icon. Ifmultiple icons are displayed, the system 2 may be configured to assumethat the user 16 has gestured toward the icon closest to the gesturinglocation 19 associated with the user's gesture.

Because, as described later in greater detail, users 16 may have one eyethat is more dominant than the other, an actual gesturing location(detected pointing location) on a display 6 may differ from the actualdisplayed location of an icon to which the user 16 believes he/she ispointing. Thus, in order to increase the accuracy of a touch-freegesture system, the system may be enabled to determine an offset Oassociated with the location of the first display information (e.g.,icon 20) and the first gesturing location 19, as is generally depictedin FIG. 17, step 317 of FIG. 3A, and as is described later in greaterdetail. The offset O may be determined and applied in a number ofdifferent ways. For example, the offset O may be accomplished throughthe selection of a “virtual eye” at a location offset from the centerpoint between the user's two eyes, and the virtual eye may be used as astarting point of the line of sight. Alternatively, the offset O may bedetermined based on detected differences on a display between an icon(e.g., icon 20) at which a user 16 is gesturing and the detectedlocation of gesturing, i.e., gesturing location 19. An example andgreater detail are provided later with respect to FIGS. 12, 13, and 17.

While this description at times refers to the determination of a singleoffset O, it is to be understood that this is a short hand and that thesystem may be enabled to determine multiple offsets (sequentially or atperiodic intervals) in order to potentially achieve a higher level ofaccuracy, as well as multiple offsets associated with differentlocations on the display 6, associated with different sizes of icons,associated with different clustering of icons, or associated with thedistance between the icons in the area or location on the display wherethe gesture location is detected. Furthermore, an offset may exist inany direction represented by up to one-coordinate, two-coordinates, orin the case of a 3-D offset, three-coordinates. Further, an Offset O maybe any mathematical representation associate with the difference of thelocation on the display on which the user perceives as pointing at, andthe gesturing location 19 as directed by the system 2. The offset O canrelate to the distance between these two locations on the display 6itself (e.g., 24 pixels to the right, and 8 pixels to the top), a vectorindicating the difference between the two locations (2-D or 3-D vector),a set of figures relating to the distance between these two locations,and/or a formula that describes the distance between these twolocations. The offset O can relate to the “virtual eye” location ascompared to other anatomical locations on the user's face (such as theuser's eye 22). The offset O can relates to the location of the user'spointing element (e.g., fingertip 14), and may be used to correct thedetected location of the pointing element. The offset O can also relateto any feature of the calculated viewing ray 24 starting front the“victual eye,” through the pointing element, and intersecting with thedisplay 6.

When second information is thereafter displayed, as is depicted in step319 of FIG. 3A, and the user 16 gestures at the second information,graphical or numerical data associated with that second gesture may bereceived via an image sensor (e.g., image sensor 8 in FIG. 1), asdepicted by step 321 of FIG. 3A, and transferred to a processor (e.g.,processor 15 in FIG. 2A). Due to the fact that the user has one eye thatis dominant over the other, the second or subsequent gesture locationmay not accurately align with the second information, i.e., the locationat which the user intended to point. Using the previously determinedfirst offset (or plurality of offsets) from step 317, however, theprocessor 15 may be configured to correct for the misalignment, as isdepicted in step 323. As the result of the correction, the user's intentmay be determined, and a corresponding action may be initiated. One suchaction may include sending an output to display 6 to provide a displayedrepresentation of the user's intent. (e.g., although the system may havedetected the user pointing 3 inches to the left of an icon, the offsetcorrection might enable a cursor to be displayed closer to or on theicon.) Exemplary operation of the offset is described in greater detailin connection FIGS. 12 and 13.

In some embodiments, the fingertip 14 does not contact the display 6while pointing or during the tapping motion. In an example illustratedin FIG. 1, the fingertip 14 is pointing towards the icon 20 b. Theprocessing unit 12 may detect the user 16 pointing the fingertip 14towards the icon 20 b for a predefined amount of time or detect the user16 performing a tapping motion (or clicking motion) towards the icon 20b by analyzing images of an image stream (e.g., video stream) obtainedby the image sensor 8. When the processing unit 12 determines that aparticular icon has been selected, a corresponding command and/ormessage associated with the selected icon may be executed or generated,respectively. The generation of a message or the execution of a commandmay address corresponding results to any one or more of an operatingsystem, one or more local services, one or more applications, one ormore devices, one or more remote applications, one or more remoteservices, or one or more remote devices.

In some embodiments, the device 4 may communicate with an externaldevice or website in response to a selection of a graphical element witha communication. The communication may include sending a message to oneor more of an application running on the external device or website, aservice running on the external device or website, an operating systemrunning on the external device or website, a process running on theexternal device or website, one or more applications running on aprocessor of the external device or website, a software program runningin the background of the external device or website, or one or moreservices running on the external device or website. Further, thecommunication may include sending a message to an application running onthe device 4, a service running on the device 4, an operating systemrunning on the device 4, a process running on the device 4, one or moreapplications running on a processor 15 of the device 4, a softwareprogram running in the background of the device 4, or one or moreservices running on the device 4. The communication with the externaldevice or website may occur over a communication network.

in response to a selection of a graphical element, a message may he sentto the external device or website requesting data relating to agraphical element identified in an image from an application running onthe external device or website, a service running on the external deviceor website, an operating system running on the external device orwebsite, a process running on the external device or website, one ormore applications running on a processor of the external device orwebsite, a software program running in the background of the externaldevice or website, or to one or more services running on the externaldevice or website. A message may also be sent to the device 4 requestingdata relating to a graphical element identified in an image from anapplication running on the device 4, a service running on the device 4,an operating system running on the device 4, a process running on thedevice 4, one or more applications running on a processor 15 of thedevice 4, a software program running in the background of the device 4,or to one or more services running on the device 4.

The message sent to the external device or website may include acommand. This command may include, for example, a command to run anapplication on the external device or website, a command to stop anapplication running on the external device or website, a command toactivate a service running on the external device or website, a commandto stop a service running on the external device or website, or acommand to send data relating to a graphical element identified in animage. The message to the device 4 may include a command. This commandmay include, for example, a command to run an application on the device4, a command to stop an application running on the device 4, a commandto activate a service running on the device 4, a command to stop aservice running on the device 4, or a command to send data relating to agraphical element identified in an image.

In response to a selection of a graphical element, the device 4 mayreceive data from the external device or website relating to a graphicalelement identified in an image, and the device 4 may present thereceived data to the user 16.

The commands executed and/or the messages generated by pointing with twopointing elements, such as two hands, can include, for example,selecting an area, zooming in or out of the selected area by moving thefingertips away from or towards each other, and rotating the selectedarea by rotating the fingertips. The command executed and/or the messagegenerated by pointing with two fingers can also include creating aninteraction between two objects, such as combining a music track with avideo track. In addition, a user 16 may execute a command and/orgenerate a message to create a gaming interaction by pointing one fingerat the display 6 and pointing and moving another finger towards anotherlocation on the display 6. In addition, a user 16 may execute a commandand/or generate a message to slide an icon across the screen (thisaction may be used for unlocking the device), scroll a page or a folder,and increase or decrease the volume. A command may be a “swipingcommand” over one or more icons.

The commands may be executed and/or messages may be generated inresponse to a predefined gesture performed by the user 16 afteridentification of a location on the display 6 at which the user 16 hadbeen pointing. The system 2 may be configured to detect a gesture andexecute an associated command and/or generate an associated message. Thedetected gestures may include, for example, one or more of a swipingmotion, a pinching motion of two fingers, pointing, a left to rightgesture, a right to left gesture, an upwards gesture, a downwardsgesture, a pushing gesture, opening a clenched fist, opening a clenchedfist and moving towards the image sensor 8, a tapping gesture, a wavinggesture, a circular gesture performed by finger or hand, a clockwiseand/or a counter clockwise gesture, a clapping gesture, a reverseclapping gesture, closing a hand into a fist, a pinching gesture, areverse pinching gesture, splaying the fingers of a hand, closingtogether the fingers of a hand, pointing at a graphical element, holdingan activating object for a predefined amount of time, clicking on agraphical element, double clicking on a graphical element, clicking onthe right side of a graphical element, clicking on the left side of agraphical element, clicking on the bottom of a graphical element,clicking on the top of a graphical element, grasping an object,gesturing towards a graphical element from the right, gesturing towardsa graphical element from the left, passing through a graphical elementfrom the left, pushing an object, dapping, waving over a graphicalelement, a blast gesture, a clockwise or counter clockwise gesture overa graphical element, grasping a graphical element with two fingers, aclick-drag-release motion, sliding an icon, and/or any other motion orpose that is detectable by a sensor.

FIG. 2B depicts an exemplary schematic representation of the system 2.In particular, the processing unit 12 may comprise, for example, animages data-repository 202 configured for storing images captured by theimage sensor 8; a graphical element generator (i.e., icon generator) 204configured to generate and display icons on the display 6; a commandexecution module 206 configured to enable execution of a commandassociated with a respective icon displayed on the display 6; acalibration module 208 configured for performing a calibration process;and a machine vision unit 210. The machine vision unit 210 may comprise,for example, a gesture recognition module 212 configured to identifygestures made by the user 16 from the captured images; and a viewing rayidentification module 214 configured to identify a point or an locationon the display 6 towards which a user 16 is pointing.

FIG. 3B depicts a flowchart showing an exemplary method that may beperformed by the system 2. The method may include a step 301, capturingat least one image of a viewing space in front of the display 6. Forexample, the image sensor 8 may capture a succession of images of theviewing space in front of the display 6. The images may be captured andstored in a data-repository, such as the image data-repository 202. Thecaptured images may be processed at a later occasion or they may beimmediately processed as they are being captured. The captured imagesmay include, for example, a user 16 positioned in front of the display 6and attempting to interact with one or more graphical element displayedon the display 6 (i.e., a user 16 attempting to point a pointingelement, such as the fingertip 14, towards an icon or text displayed onthe display 6).

The captured images may also include one or more predefined parts of theface of the user 16, such as one or more of the user's eyes 22, and oneor more pointing elements, such as the fingertip 14. The method mayinclude a step 303, detecting a pointing gesture in the at least oneimage. In particular, identification of the user 16 and the pointingelements may be accomplished, for example, with the help of the machinevision unit 210, including the gesture recognition module 212.

The method may include a step 305, identifying a point or location onthe display 6 towards which the one or more pointing elements arepointing. This point may be identified, for example, with the help ofthe viewing ray identification module 214. Since the user's eye 22, thefingertip 14, and the point on the display 6 that a selected icon islocated may be collinear and lie on a common viewing ray 24, theprocessing unit 12 may be configured to identify the viewing ray 24. Thestep 305 may further include one or more processes designated forenhancing the identification of the point on the display 6 towards whichthe user 16 is pointing. Although FIG. 1 depicts the user's eye 22 as apart of the face of the user 16 that may be used for identifying aviewing ray 24, other parts of the face may similarly be identified andused for that purpose. Thus, wherever one or more of the user's eyes 22are mentioned in the present description it should be considered as anexample only and should be broadly interpreted to include other parts ofthe face as well.

The method may include a step 307, identifying a visual object displayedon the display 6 which fulfils a predefined criterion with respect tothe identified point on the display 6. For example, the visual object(or graphical element) may be the closest visual object to theidentified point on the display 6 that is selected by the user 16.

The method may include a step 309, determining when a predefinedcondition is met while the pointing element is pointing towards thevisual object. Step 309 may be performed after a selected visual object(graphical element) has been determined. As mentioned above, thecondition may include, for example, a predefined period of time duringwhich the pointing element is pointing towards the determined pointand/or a predefined gesture (e.g., a tapping motion towards the display6), or any other suitable condition.

The method may include a step 311, executing a predetermined operationcommand associated with the visual object. Step 311 may be performedafter the predefined condition has been met. For example, the commandexecution module 206 may be configured to execute a command, such asinteracting with other functional elements in device 4 that areassociated with the selected graphical element.

In order to determine which graphical element is selected by aninteracting user 16, it is desirable to first identify a point on thedisplay 6 towards which the user 16 is pointing.

In some embodiments, the image sensor 8 may include a 3-D image sensoror a pair of 2-D stereoscopic image sensors. In these embodiments, theprocessing unit 12 may be configured to locate x-, y-, and z-coordinatesof the user's eye 22 and/or the fingertip 14 in the 3-D viewing space.The processing unit 12 may then determine a viewing ray 24 having avertex at the eye 22 and passing through the fingertip 14. Asillustrated in FIG. 1, a dot or marker 18 may be positioned on thedisplay 6 at a location where the viewing ray 24 intersects the display6. In some embodiments, the system 2 may comprise a combination of 2-Dand 3-D image sensors and may be configured to obtain x- andy-coordinates of the user's eye 22 and the fingertip 14 from imagescaptured by the 2-D image sensor and obtain the z coordinate of theuser's eye 22 and the fingertip 14 from images captured by the 3-D imagesensor. In some embodiments, the 3-D image sensor may be configured toobtain low-resolution images. The combined information from the 2-D and3-D image sensor may be used for determining a 3-D viewing ray 24.

In other embodiments, the image sensor 8 may include a 2-D positionsensor. A user 16 standing in front of the display 6 may point anywhereon the display 6 and the specific viewing ray 24 associated with theuser's pointing may be determined based on the 2-D position (i.e., thex- and y-coordinates) of the user's eye 22 (or other anatomicallocation) and the 2-D position of the pointing element (i.e., thefingertip 14) as captured by the image sensor 8 along with the distancebetween the anatomical location and the display 6 and the distancebetween the pointing element and the display 6 or the user 16 or user'seye 22.

In some embodiments, the processing unit 12 may be configured todetermine a point on the display 6 towards which the user 16 ispointing. For example, the x- and y-coordinates of the user's eye 22 maybe determined form the images captured by the image sensor 8. Asillustrated in FIG. 6, for example, the distance R1 between the user 16and the display 6 can be estimated. Based on the estimated distance R1,the z-coordinate of the user's eye(s) 22 in the viewing space can beobtained.

The distance R1 can be determined, for example, based on the distancebetween the user's eyes 22 as identified in an image obtained by theimage sensor 8. A similar distance exists between the eyes (e.g. pupils)of different individuals. The degree of similarity is higher forindividuals belonging to the same ethnic group. Therefore, the distancebetween the eyes of a user 16 may be indicative of the distance R1between the user's eye 22 and the display 6.

The x- and y-coordinates of the pointing element can be determined fromimages captured by the image sensor 8. Using the estimated distance R1between the pointing element and the user 16, the z-coordinate of thepointing element in the viewing space can be obtained.

Once the x-, y-, and z-coordinates of the user's eye 22 (or virtual eyeor other anatomical feature) and the x-, y-, and z-coordinates of thepointing element have been obtained, a straight line (viewing ray 24)extending through these two points may be determined and the point wherethis line intersects the display 6 may be identified. The identifiedpoint of intersection of this line with the display 6 may be determinedas the point on the display 6 towards which the user 16 is pointing,e.g., Gesture Location.

In some embodiments, the display 6 may be a 3-D display and thedisplayed graphical elements may be 3-D graphical elements that areperceived to the user 16 as being located in the viewing space locatedin front of the display 6 or behind the device 4. Thus, determining thelocation of the point towards which the user 16 is pointing can beexecuted with a 3-D display, wherein the identified viewing ray 24extends from the user's eye 22 through the pointing element and to avisual element representing a respective graphical element that isperceived to the user 16 as being in the space in front of the display6.

In some embodiments, the display 6 is a 2-D display and the displayedgraphical elements may be 2-D graphical elements. Thus, determining thelocation of the point towards which the user 16 is pointing can beexecuted with a 2-D display 6, wherein the identified viewing ray 24extends from the user's eye 22 through the pointing element and to theintersection of the viewing ray 24 with the display 6.

FIG. 4 depicts an exemplary graphical representation of a user 16pointing towards the display 6. To better determine the point on thedisplay 6 towards which a user 16 is pointing, it may be advantageous toidentify the location of the pointing element in the viewing space whilepointing. Therefore, it may be advantageous to identify a point which isthe location of the pointing element during a pointing gesture that thepointing element is closest to the display 6. The hand motion of theuser 16 during the pointing gestures is illustrated as a motion vectorcharacterized by the point P′ located closest to the display 6.

The system 2 may be configured to identify the point P′ during apointing gesture, in some embodiments, wherein the image sensor 8includes a 3-D image sensor or a stereoscopic image sensor, theprocessing unit 12 may be configured to extract the point P′ from acaptured image, such that the z-coordinate of point P′ may indicate theposition of the pointing element when the pointing element is locatedclosest to the display 6.

In some embodiments, wherein the image sensor 8 includes a 2-D imagesensor, the point P′ may be identified based on an analysis of one ormore features of the pointing element during the pointing gesture, whichmay include, for example, the motion path of the pointing element, themotion vectors of the pointing element, the change in motion vectors ofthe pointing element, the pose of the pointing element, the size of thepointing element, the acceleration and/or deceleration of the pointingelement. In some embodiments, identification of point P′ can be based ona change in the position of the pointing element in the viewing plane.For example, after the user 16 extends his arm and the point P′ has beenreached, the user 16 may retract his arm. Changes in the position (i.e.,coordinates) of the pointing element in the viewing space, may indicatethat the arm has been retracted and may be used to identify point P′.

In some embodiments, the point P′ can be identified based on changes inthe size of the pointing element during the pointing gesture. Forexample, as the user 16 extends his arm towards the display 6, the sizeof the pointing element may increase (e. g,, the size of the fingertip14 in the captured images may increase). The processing unit 12 may beconfigured to track the size of the pointing element and determine thelocation of the pointing element, at which the size of the pointingelement is the largest.

FIG. 5 depicts a flowchart illustrating an exemplary method foridentifying point P′. Operations described with reference to FIG. 5 canbe executed, for example, by the processing unit 12 including themachine vision module 210.

The method may include a step 501, capturing at least one image of theuser 16 interacting with the display 6. In particular, the image sensor8 may capture images of the user 16 while attempting to interact withicons displayed on the display 6. The method may include a step 503,detecting a pointing element in the at least one image. The method mayinclude a step 505, identifying changes in size of the tip of thepointing element. During step 505, the captured images may be processedand changes in the size of the pointing element in different images maybe identified. For example, the ratios between the size of the fingertip14 at an initial position, at the start of the pointing gesture, withrespect to the size of the fingertip 14 along different positions alongthe motion of the arm towards the display 6 may be calculated and apercentage of growth in size of the fingertip 14 can be identified. Thelargest size of the pointing may occur once the pointing element ispositioned at the shortest distance from the display 6, and the greatestchange in its size with respect to its initial position may be recorded.When the arm is then retracted, the change in size may decrease as thesize of the pointing element becomes smaller.

The method may include a step 507, identifying a position of thepointing element where the greatest change is determined between theinitial size and the current size of the tip of the pointing element.The method may include a step 508, designating the identified positionas point P′.

FIG. 6A is a top view illustration depicting a user 16 pointing towardsa display 6. As shown in FIG. 6A, the image sensor 8 may be positionedadjacent to the display 6, and may be capable of capturing images of a3-D viewing space indicated by the broken lines 10. The distance R1 ofthe user 16 from the display 6 is indicated by line R1, which extendsfrom the display 6 to a point on an imaginary line between the eyes(e.g., between the pupils) of user 16. As mentioned above, the distanceR1 can be determined, for example, based on the distance between theuser's eyes as identified in an image obtained by the image sensor.

Furthermore, based on typical human behavior it can be assumed that whena user 16 is pointing towards the display 6 his arm may not necessarilybe fully extended from his body, and it can be assumed that the arm isnot necessarily fully retracted when it returns to the body. Theextension length of the pointing element from the body of the user 16towards the display 6 is indicated in FIGS. 6A and 66 by a range R2.

The range R2 can be estimated, for example, based on the user's heightand the estimated proportions of his body. In addition, the range R2 maybe estimated based on information indicating the distance R1 of the user16 from the display 6. Additionally, information with respect to the ageand gender of the user 16 may also be used during the estimation of therange R2. In addition, the range R2 may be estimated from recodedstatistical information obtained from a plurality of users 16 thatindicates the extension of the arm of the users 16 while pointing (whichmay also take into consideration the height and proportions of each user16). Further, this information can be classified based on geographicaland/or ethnic origin of the users 16, in order to differentiate betweendifferent pointing gestures that may be associated with differentgeographical and/or ethnic origins. Thus, for example, an average lengthfor range R2 can be calculated from the recorded information withrespect to the arm extension lengths of many users 16.

While the range R2 may enable estimation of the distance of the pointingelement from the user's body, some deviation may exist between theextensions of the arm in the pointing gestures made by different users16. FIG. 6A also illustrates deviation Δr, which represents a potentialdeviation in the extension of the arm between different pointinggestures. Assuming, as mentioned above, that range R2 is an averagevalue of the measured extension of the arm in pointing gestures ofdifferent users 16, deviation Δr can be determined, for example, as avalue which equals one or more standard deviations of that average valuedescribed above (e.g., after normalization of the height factor).

FIG. 6B is a top view illustration depicting different exemplaryextensions of a pointing element. As shown in FIG. 6B, an image sensor 8may be positioned adjacent to display 6 and may be capable of capturingimages of a 3-D viewing space. Images of the user 16 may be captured bythe image sensor 8. Unlike images captured by a 3-D image sensor, imagesthat are captured by a 2-D image sensor may provide an image projectionof the user 16 onto the display 6 without depth perception. As show inFIG. 6B, the user 16 may extend his arm and point towards the display 6.Two extensions having different lengths are shown. A viewing rayextending from extension a (the longer extension) intersects the display6 at a point P1, and a viewing ray extending from extension b (theshorter extension) intersects the display 6 at point P2.

Although, extensions a and b may be pointing to different locations onthe display 6, both extensions may be viewed by the image sensor 8 inthe same manner, i.e., converging on a same line R3.

As mentioned above, in order to identify the gesturing location 19, (thelocation on the display 6 towards which a user 16 is pointing), aviewing ray 24 connecting the eyes of the user 16, the pointing element,and a respective point of intersection of the viewing ray 24 with thedisplay 6 may be determined. However, as a result of variousuncertainties that may exist during the identification of the viewingray 24, uncertainly may also exist as to where exactly on the display 6the user 16 is pointing. These uncertainties may include, for example,uncertainty with regard to the identification of the tip of the pointingobject, and with regard to the identification of a point located betweenthe eyes of the user 16, which may best represent the viewing ray 24.Furthermore, as discussed above, 2-D image sensors may includeadditional uncertainty with respect to the actual extension of thepointing element from the body of the user 16 and with respect to thedistance of the user 16 from the display 6.

As a result of these potential uncertainties, when a user 16 is pointingtowards the display 6, instead of identifying a single point on thedisplay 6 towards which the user 16 may be pointing, a larger area (orcandidate plane) may be identified on the display 6. The candidate planemay represent area, in which a plurality of possible lines extendingfrom the user's eye 22 pass through the pointing element and intersectthe display 6. An image obtained by image sensor 8 may include theuser's eye 22 and the fingertip 14, and may include a projection of theviewing ray 24 onto projection plane (the projection plane being theplane captured by the image sensor 8). The set of all lines in theviewing space whose projection onto the projection plane is the same asthe projection of the viewing ray 24 onto the projection plane forms thecandidate plane. Each line extending from the user's eye 22 through apointing element and a common candidate plane is a candidate viewingray. A candidate plane can be characterized by different featuresincluding, for example, the number of pixels in a candidate plane (acandidate plane can comprise one or more pixels), its size (e.g., inpixels or its measured diameter), its shape, its location on the screen,etc.

Accordingly, it would be advantageous to more accurately identify pointor location on the display 6 towards which a user 16 is pointing,notwithstanding the uncertainties mentioned above. Operations describedwith reference to the following figures are applicable in systemequipped with any one of 2-D, 3-D, and stereoscopic image sensors.

In some embodiments, the user 16 may perform a calibration procedure. Inthe calibration procedure, the user 16 may be instructed to pointsequentially at one or more graphical elements (e.g., icons) displayedon the display 6. One of the graphical elements may include, forexample, a graphical element, such that a selection thereof may switchthe device 4 from a standby mode to an active mode or the selection mayunlock the display 6. When a user 16 points towards such a graphicalelement, the processing unit 12 may determine the position of thepointing object and the user's eye 22 in an image obtained by the imagesensor 8 while the user 16 is pointing at the graphical element. Thefingertip 14 may be identified in an image obtained by the image sensor6, for example, as disclosed in Applicant's co-pending U.S. patentapplication Ser. No. 10/593,628, which published as U.S. patentpublication no. 2008/0042981, and which is hereby incorporated byreference in its entirety.

Methods for identifying a face and eyes in an image are known in theart. When the user 16 subsequently points to an icon on the display 16,the processing unit 12 may determine the location on the display 6, andhence the icon, to which the user 16 is pointing using calibration data,together with limitations on the distance between the user's eye 22 andthe fingertip 14 and the distance between the user 16 and the display 6,which may be determined from the distance between the user's eyes in theimage obtained by the image sensor 8.

FIG. 7 depicts a flowchart illustrating an exemplary method forperforming a calibration process with the processing unit 12 includingthe calibration module 208. During the calibration process, the distanceR1 between the user 16 and the display 6 and the range R2 between thepointing element and user's body may be more accurately estimated. Themethod includes a step 701, displaying on a display 6 a calibrationsequence. During step 701, one or more graphical elements (e.g., icons)may be displayed on the display 6 in a predefined order or shape and mayprovide a calibration sequence. The calibration sequence may becharacterized by the location of icons on the display 6 and the timeand/or the sequence of interaction with the different icons. The user 16may be requested to interact with the icons (e.g., point to or swipe)according to a predefined order. In some embodiments, the calibrationsequence may also include an icon to be swiped or moved according to apredefined pattern or shape.

In some cases, the user 16 may be unaware of the calibration process.For example, as mentioned above, the calibration process can bepresented to the user 16 as an interaction for unlocking the display 6(e.g., swiping an icon displayed on the display 6 from left to right).Or calibration may take place during ordinary usage as the user 16points to displayed icons.

The method may include step 703, capturing at least one image of theuser 16 interacting with the display 6. During step 703, the user 16 mayinteract with the display 6 as requested by the calibration sequence andthe image sensor 8 may capture images of the user 16 while doing so. Themethod may include step 705, determining the position of the user 16 inthe viewing plane. During step 705, the location (i.e., x- andy-coordinates) of the user 16 (e.g., the user's eye 22) in an imagecaptured by the image sensor 8 may be determined. The method may includestep 707, determining a respective point P′ of a pointing element.During step 707, a pointing gesture may identified and a point P′, whichindicates the location of the pointing element (i.e. x- andy-coordinates) where the pointing element is located closest to thedisplay 6 may be determined.

The method may include a step 709, determining a viewing ray 24connecting a respective icon on the display 6 to the 2-D location of thepoint P′, and to the 2-D location of the user 16. Since the calibrationprocess may be executed in accordance with a predefined calibrationsequence, information indicating which of the graphical elementsdisplayed on the display 6 that the user 16 is currently requested to bepointing, is available to the processing unit 12. Therefore, theprocessing unit 12 can more easily correlate between the viewing ray 24connecting the location of the user 16, the point P′ and a respectivegraphical element on the display 6, towards which the user 16 ispointing. Accordingly, a respective viewing ray 24 may extend from therelevant graphical element on the display 6, through the pointingelement located at the respective point P′ and to the respectivelocation of the user 16.

FIG. 8 is a top view illustration depicting an exemplary calibrationprocess. As shown in FIG. 8, an image sensor 8 may be positionedadjacent to the display 6 and may be capable of capturing images of a3-D viewing space indicated by the broken lines 10. A user 16 may pointtowards an icon G₁, which is part of a calibration sequence (not shown).The 2-D position of the user 16 standing in front of the display 6 maybe located on a line L1. The distance R1 between the user 16 and thedisplay 6 can be estimated as described above; however, the calibrationprocess may enable improvement over this estimation. The 2-D location ofthe point P′ may be located on a line L2. The range R2 extending betweenthe line L1 and the line L2 can be estimated as mentioned above;however, the calibration process may enable improvement over thisestimation.

As described above with reference to FIG. 6A, uncertainty may exist withregard to the estimated range R2, as indicated above by the deviationΔr. Thus, the distance between the line L1 and the line L2 can beestimated to be anywhere between R2+Δr/2 and R2−Δr/2. During thecalibration process, a first straight line L3 may extend from the iconG₁, on the display 6 and connect to the lines L1 and L2, such that thedistance between the line L1 and the line L2 equals R2−Δr/2; and asecond straight line L4 may extend from the icon G₁ on the display 6 andconnect the line L2 and the line L1, such that the distance between theline L1 and the line L2 equals R2+Δr/2. As shown in FIG. 8, the distancebetween the point a on the line L2 and a point u1 on the line L1 equalsR2−Δr/2, and the distance R2″ between the point b on the line L2 and apoint u2 on the line L1 equals R2+Δr/2.

A point EL or the line L1 may represent the estimated location of theuser 16, which may be determined to be any point located between thepoints u1 and u2. For example, the point EL can be the point locatedexactly at the middle between the points u1 and u2. A respective pointon the line L2 can be determined by extending a straight line from theselected point on the line L1, through the line L2, and to the icon i.The calibration method may include a step 711, providing feedback to theuser 16 and repeating steps 703 to 709 as the user 16 follows thecalibration sequence. Feedback can be provided to the user 16 by, forexample, changing one or more of the color, the size, the shape, and theposition of the respective graphical element or by an audibleindication. As the user 16 follows the calibration sequence andinteracts with the respective graphical elements on the display 6, thesteps 703 to 709 may be repeated.

For any icon G₁, a respective point ELi on the line L1 representing thelocation of the user 16 can be determined as described above. The methodmay include a step 713, determining the ranges R1 and R2 based onmultiple viewing rays generated with different visual elements. Duringstep 713, a final estimation of ranges R1 and R2 may be determined. Thefinal estimation of ranges R1 and R2 can be determined based on all thepoints EL obtained with respect to the different icons in thecalibration sequence (e.g., a point located at the center of allidentified points).

The method may include a step 715, responsive to an identified pointinggesture towards the display 6 identifying a respective viewing ray 24based on the ranges R1 and R2. Step 715 may occur once the system hasbeen calibrated and/or during regular operation.

The processing unit 12 may be configured to determine a different rangeR2 for each icon on the display 6 and may create a map associating thedifferent icons and their respective locations on the display 6 with acorresponding range R2. Any other point on the display 6 (not populatedby a calibration icon) can be associated with the respective range R2based on the map. This association can be based, for example, on alinear combination of the location of icons and their respective rangesR2.

During regular operation, the processing unit 12 can be configured,responsive to a pointing gesture of a user 16 towards the display 6, toidentify the general direction of the pointing gesture, and to select aspecific range R2 that is associated with a calibration icon located ata point on the display 6, which is closest to the identified generaldirection.

In addition to or instead of the calibration process described above,other techniques may be possible to more accurately identify a point orlocation on the display 6 towards which the user 16 is pointing. Asexplained above, due to different uncertainties related to parametersused when determining a point on the display 6 towards which the user 16is pointing, a candidate plane may be identified on the display 6, andmay represent a plane, on which a plurality of possible lines extendingfrom the eyes of the user 16, passes through the pointing element, andintersects with the display 6.

FIG. 9 depicts a flowchart illustrating an exemplary method, which mayemploy the processing unit 12 including the machine vision unit 210, forreducing the size of the candidate plane and to more accurately identifythe point the display 6 towards which the user 16 is pointing.

The method may include a step 901, capturing at least one image of theuser 16 interacting with the display 6. During step 901, the imagesensor 8 may capture images of the user 16 attempting to interact withgraphical elements displayed on the display 6. The method may includestep 903, identifying the user's eye 22 and the pointing element in theat least one image. In particular, the captured images may be processedand the user's eye 22 and the pointing element may be identified in thecaptured images. The method may include a step 905, obtaining a sequenceof position data. During step 905, the motion path of the pointingelement may be traced while it is being extended towards the display 6and a sequence of position values comprising different positions of thepointing element (e.g., the fingertip 14) in the motion path may beobtained.

The position values include parameters that may describe the position ofa pointing element in a given image. For example, position values can bex- and y-coordinates of the tip of the pointing element in the viewingplane captured by the image sensor 8. The user 16 may extend his armcloser towards the display 6 and the 2-D position of the fingertip 14with respect to the viewing plane (indicated by x- and y-coordinates)can be obtained and stored.

While the arm is being extended closer towards the display 6 during apointing gesture, starting from a certain distance from the display 6and moving towards a convergence point and onward, the pointing elementmay point to the same location on the display 6 (e.g., the pointingelement may be aligned on a similar viewing ray 24). During this motionpath, the image sensor 8 may capture a sequence of images, such thatdifferent points of the pointing element that may be aligned on the sameviewing ray 24 that passes through the user's eye 22, the pointingelement (i.e., the fingertip 14), and the point on the display 6.

The method may include step 907, identifying the pointing element at thepoint P′. During step 907, the motion of the pointing element (i.e.,fingertip 14) may be traced until it has been identified that thepointing element is located at the point P′. At this point, a predefinednumber of position values may be selected from the sequence of positionvalues extracted from the motion path of the pointing element towardsthe display 6. The selected position values can be for example, theposition values that correlate with the N images (or frames) before thepointing element has reached the point P′, which is the closest point tothe display 6. Alternatively, the selected position values can besequential positions located on similar viewing rays.

As discussed above, each position value in the sequence of positionvalues (represented by, for example, x- and y-coordinates on the viewingplane) may be associated with a respective viewing ray 24 connecting theuser's eye 22, through the pointing element located at a positionindicated by the respective position value, and intersecting the display6. Because of the uncertainties discussed above, each of the viewingrays may be associated with a respective candidate plane on the display6.

The method may include a step 909, selecting a predefined number ofposition data components and identifying their respective candidateplanes on the display 6. During step 909, the respective candidateplanes may be associated with the selected position values identified onthe display 6. The method may include a step 911, identifying anoverlapping region between all of the candidate planes as the selectedarea on the display 6. During step 911, the overlapping region betweenthe candidate planes may be identified and designated as the selectedviewing plane, thereby reducing the size of the candidate plane andidentifying a smaller area on the display 6 as the point toward whichthe user 16 is pointing.

FIG. 10 depicts an exemplary Venn diagram showing three partiallyoverlapping candidate planes. Each of the three candidate planes isillustrated as a circle covering a certain area or location on thedisplay 6. Each plane is associated with a different position value ofthe three last position values in a recorded sequence of position values(position values: n, n−1, and n−2).

As discussed above, from the convergence point and onward, each of thethree different viewing planes that are associated with differentposition values in the sequence of position values may cover anoverlapping area. As illustrated in FIG. 9, the three candidate planesmay point to the same area and share an overlapping portion (shown witha crosshatching pattern), which is designated as the selected projectionplane.

FIG. 11 depicts a flowchart illustrating an exemplary method that may beperformed with the processing unit 12. The system 2 may be configured toprovide feedback to a user 16 pointing towards the display 6. Thisfeedback may assist the user 16 to direct the pointing element towards adesired point on the display 6 and to more easily select a desiredgraphical element. The following operations associated with FIG. 11 areprovided in the context of a 2-D display, however similar operations maybe executed with a 3-D display.

The method may include a step 1101, capturing at one image of a user 16interacting with the display 6. During step 1101, the image sensor 8 maycapture images of the user 16 while attempting to interact with iconsdisplayed on the display 6. The method may include a step 1103,identifying the user's eye 22 and the pointing element in the at leastone image. The method may include a step 1105, displaying a pointingvisual effect on the display 6. The step 1105 may occur responsive tothe detection of a pointing gesture. The pointing visual effect can be,for example, a smudge of color characterized by any shape or color,transparent or opaque, and located at any location with respect to thegraphical elements that are displayed on the display 6, which may be ontop of or behind the graphical element. For example, the pointing visualeffect may have one or more of a round symmetrical shape, a transparentcolor through which the graphical elements are discernible, and appearto the user 16 as a flashlight beam illuminating a portion of thedisplay 6. The initial location of the pointing visual effect on thedisplay 6 can be determined based on the initial pointing direction ofthe pointing element.

The method may include a step 1107, tracing the motion path of thepointing element. While tracing the pointing element, a respectivesequence of position values can be obtained as described above withreference to step 905 and shown in FIG. 9.

The method may include step 1109, altering one or more characteristicsof the pointing visual effect in correlation with the movement of thepointing element. For example, as the pointing element is extendedcloser towards the display 6 one or more characteristics of the pointingvisual effect may be altered in correlation with the movement of thepointing element. The characteristics can include, for example, one ormore of the shape, size, and color of the pointing visual effect. Forinstance, the pointing visual effect can decrease in size as thedistance between the pointing element and the display 6 decreases.

As the user 16 extends his arm towards the display 6, the user 16 mayintuitively attempt to move the pointing visual effect towards theposition of a desired graphical element displayed on the display 6. Therectifying movements of the pointing element, which may be made by theuser 16 in an attempt to move the pointing visual effect towards theselected graphical element, may be identified by the processing unit 12.Optionally, the position of the pointing visual effect on the display 6can change based on the rectifying movement of the user 16. For example,the position of the pointing visual effect on the display 6 can changein correlation with the direction of the rectifying gestures that aremade by the user 6.

In another example, a circular pointing visual effect may become smalleras the pointing element is brought closer towards the display 6.Optionally the pointing visual effect can decrease in the direction ofthe rectifying gesture. FIG. 16A illustrates a pointing visual effectwhich has decreased in size and has migrated to the left in response toa rectifying gesture towards an icon (indicated by the square) locatedon the left and FIG. 16B illustrates a pointing visual effect which isdecreased in size and migrates to the right in response to a rectifyinggesture towards an icon (indicated by the square) located on the right.

The method may include a step 1111, determining, based on the knownposition of the pointing visual effect on the display 6 and on themovement of pointing element, a graphical element. For example, an iconthat complies with a predefined criterion with respect to the pointingvisual effect can be identified as a selected icon. Furthermore, an iconthat is located closest to the smudge or located relative to the smudgein the direction of the rectifying gesture can be determined as the iconselected by the user 16.

As discussed above, the processing unit 12 may be configured to identifya viewing ray 24 extending from the user's eye 22, passing through thetip of the pointing element, and then passing on to the display 6. Tothis end, the processing unit 12 may be configured to identify theuser's eye 22 in the captured images.

FIG. 12 depicts a flowchart illustrating an exemplary method forcomputing a viewing ray 24 from the user's eye 22 through a tip of apointing element to a desired location on the display 6 to which theuser 16 points by taking into account which one of the user's eyes isdominant when the user 16 points at the display 6. When pointing at agraphical element on the display 6, eye dominance may cause there to bean offset between an actual location of an icon on the display 6 and alocation where the user 16 points, in other words, when computing aviewing ray 24 from a mid-point between the user's eyes through a tip ofthe pointing element, the resulting viewing ray may not intersect with alocation of the icon on the display 6. Thus, a first gesturing locationdiffers from a location of the first display information at least inpart as a result of one eye of the user 16 being dominant over anothereye of the user 16. To correct for this offset, a methods and systemsconsistent with the invention may compute the viewing ray 24 by using apriori information related to the user's dominant eye. This dominanceinformation may be derived by the system during a calibration or duringregular use as the system may know the precise location of a displayedicon, and therefore can determine an offset when a user 16 who isprompted to point at the icon, is detected to be pointing to a locationoffset from the icon.

A priori Information may also be obtained from a database or from theuser's conduct, such as during winking. The user 16 may be askeddirectly for such information, or the user 16 may be asked to wink.During winking, the dominant eye may be revealed as the eye that remainsopen. Accordingly, the system 2 may compute the viewing ray 24 accordingto the user's dominate eye and establish with higher accuracy, anestimate of the location that the user 16 points during the differentpointing gestures described above.

Moreover, a viewing ray 24 may be extracted from the point on thedisplay 6 to which the user 16 points through the fingertip 14. However,it is possible that it may not reach one of the user's pupils, butinstead reach a point somewhere between the two pupils that is relatedto the more dominant “pointing eye.” As discussed above and depicted inFIG. 14 below, due to an offset associated with a dominant eye, theviewing ray 24 or line of sight extends from an offset position (virtualeye V_(e)) relative to the icon, through the pointing element, and to aposition between the pupils on the user's face. Therefore, it may beadvantageous to establish a more accurate starting point between theuser's eyes from which to compute the viewing ray 24, rather thananchoring a starting point from one of the user's pupils.

However, in reality, the viewing ray 24 may intersect any point locatedon an imaginary line extending between the two eyes of the user 16, at apoint which can overlap with either the left eye or the right eye or anyother point therebetween. In order to more accurately determinegesturing location 19, the point on the display 6 to which a givenviewing ray 24 extends, it might be advantageous to determine a moreaccurate point that represents the “virtual eye” of the user 16corresponding to a viewing ray 24 associated with an intended displaylocation to which the user intends to point, rather than arbitrarilychoosing one of the user's right or left eyes or any point on the linethat connects them.

In more detail, FIG. 12 depicts an exemplary method for determining areference point between the user's eyes with the processing unit 12including the calibration module 208. The reference point may representa more accurate intersection point of a given viewing ray 24 with a lineof sight extending between the eyes (e.g., pupils) of the user 16. Theprocess described with reference to FIG. 12 can be performed, forexample, as a onetime calibration process or may be updatedperiodically.

The method may include a step 1201, displaying an icon on the display 6.The method may include a step 1203, capturing images of the user 16interacting with the display 6. The images captured in step 1203 may becaptured by the image sensor 8.

Optionally the user 16 may be instructed or prompted to point towardsthe icon displayed on the display 6. The instructions can be provided tothe user 16 on the display 6, as an audible instruction, as part of thecalibration process, or simply as a display of information requiringinteraction. The method may include a step 1205, identifying a firstcandidate plane generated by a viewing ray connected to a first eye ofthe user 16. During step 1205, a pointing gesture of the user 16 towardsthe display 6 may be identified and a first viewing ray connecting oneof the user's eyes, through the pointing element, and a first gesturinglocation on the display 6 may be determined and a respective candidateplane may be identified on the display 6.

The method may include a step 1207, identifying a second candidate planegenerated by a viewing ray connected to a second eye of the user 16.During step 1207, a pointing gesture of the user 16 towards the display6 may be identified and a second viewing ray connecting the other one ofthe user's eyes, through the pointing object, to a second gesturinglocation on the display 6 may be determined and a respective candidateplane may be identified on the display 6. As discussed above, the firstand second candidate planes can be determined based on the methodsdescribed above, but in particular, the first and second candidateplanes can be determined, in part, based on the distance R1 between theuser 16 and the display 6 and the estimation of extension range R2between the pointing element and the location of the pointing element inthe viewing plane.

FIG. 13 is a schematic illustration showing calibration icon CI, whichmay be displayed on display 6. FIG. 13 further illustrates two candidateplanes, of which, one may be generated by a viewing ray connecting theleft eye of the user 16 (indicated by the letter L) and the other may begenerated by a viewing ray connecting the right eye of the user 16(indicated by the letter R). Note, that in the illustrated example shownin FIG. 13, a portion of candidate plane R overlaps the calibration iconCI, unlike the candidate plane L.

Based on the location of each of the two candidate planes with respectto the location of the icon, the processing unit 12 can be configured todetermine which of the two eyes is more dominant and may determine theoffset 0 associated with the gesturing location 19. The system 2 may usethat eye, or “the virtual eye” while determining additional gesturinglocations. The processing unit 12 may be configured to indicate throughinformation pertaining to the offset O, which eye is more dominant.Furthermore, the processing unit may be configured to indicate a degreeof dominance between the left and right eyes, which may be based on, forexample, a percentage of overlap of the candidate planes L and R withthe calibration icon CI and/or relative position of the “virtual eye”with respect to the user's eyes, i.e., a ratio of the distance between a“virtual eye” ye and one of the left and right eyes of the user withrespect to the total distance between the left and right eyes of theuser. The method disclosed with respect to FIG. 12 may also include astep 1209 of extending a line connecting the first candidate plane andthe second candidate plane. According to one example, illustrated withreference to FIG. 12, a line 130 (as shown in FIG. 13) can be extendedbetween the center of candidate plane L and the center of candidateplane R. The method may include a step 1211, determining a point alongthe 130 line (or close to the line 130, i.e., in the general vicinity)that is closest to the icon CI. For example, a third point on the line130 located closest to the center of the calibration icon CI (e.g.,which may be the target that the user 16 points towards), is identified.The point closest to the calibration icon CI may divide the line 130into segments A and B. In the example shown in FIG. 13, the segment Amay extend from the point closest to the calibration icon CI to thegesturing location 19 determined by calculating a line of sight“starting” from the left eye of the user 16, toward the pointing element(e.g., fingertip 14) and intersecting with the display 6. The gesturelocation may be also defined as the center of the plane L. Also, thesegment B may extend from the point closest to the calibration icon CIto the center of the gesturing location determined by calculating a lineof sight “starting” from the right eye of the user 16, toward thepointing element (e.g., fingertip 14) and intersecting with the display6. The gesture location may be also defined as the center of the planeR. The method may include a step 1213, identifying the correspondingpoint Ve (or a “virtual eye”) on the line extending between the eyes orthe pupils of the user 16 (as shown in FIG. 14).

The point Ve may divide the line extending between the eyes of the user16 into segments C and D. For example, segment C may extend from thepoint Ve to the user's left pupil, and segment D may extend from thepoint Ve to the user's right pupil. The position of the point ye on theline extending between the eyes or the pupils of the user 16 may bedetermined by interpolating the position of the point Ve, such that theproportion of A/B equals the proportion of C/D.

Processing unit 12 may be configured to use be corresponding point Ve asthe point of intersection with the face of the user 16 when identifyingviewing rays. It should be noted that this process can take place everytime a calibration takes place, to reach more collected values in orderto fine tune the location of the “virtual eye,” and to fine tune itaccording to different locations of the user 16 with respect to thedisplay 6

Optionally, the operations described with reference to FIG. 12 can berepeated with a plurality of icons, each of them located at a differentposition on the display 6. Accordingly, a different respective point Vebetween the eyes can be identified for processing pointing gesturesdirected to different areas or locations on the display 6.

The “virtual eye” location and/or the “virtual eye” mapping information,can be stored in memory (on the device 4, or external to the device 4)associated with every specific user 16 to be used in later time bysystem 2, or by other devices using the system 2. When the system 2identifies the user 16 (for example, by face recognition, user voicesignature, or expressly identified by the user 16) the system 2 mayretrieve the stored information indicating the location of the “virtualeye” and/or the “virtual eye” mapping information. The “virtual eye”and/or the “virtual eye mapping information” that is stored for eachuser 6 may include unique identity information obtained by the imagesensor 8, and may include information regarding offsets associated witheach user 6. For example, the stored information for each user 6 mayinclude facial information including the location of facial features,such as the user's eyes, including the user's pupils. In addition, theoffset information saved for each user, may include offset informationof the user 6 with respect to the user's eyes, including the user'spupils.

Embodiments of the invention may include storing one or more offsets,which are associated with vectors representing the difference on thedisplay 6 between a gesturing location 19 and the location on thedisplay 8 the user 16 is perceived to be pointing. For example, thedistance from the center of candidate plane R and the center of the iconthe user 16 was prompt to point at. Then, when the user 16 performssubsequent touch-free gestures, the system 2 may use the stored offsetsto correct the gesturing location 19 in order to correctly determine thedisplayed information (icon) at which the user is pointing. The system 2may store multiple offsets associated with different regions of thedisplay, and may use an appropriate one of the stored offsets accordingto the gesturing location 19. Embodiments of the invention may enableidentification of the pointing element of a user 16, such as thefingertip 14. Features of a hand may help identify the fingertip 14 orany other pointing element that may be held in the hand, or worn on thehand. FIG. 15 depicts a hand of a user, including the fingertip 14, afingernail 26 on the fingertip 14, and an index finger knuckle 28.

A method for determining a pointing element may include determiningwhether the pointing element is a part of or associated with the user'sright or left hand. From images captured by the image sensor 8, it mayreadily be determined which hand the user 16 moves when performing agesture. The method may include a step of determining the fingernail 26on the fingertip 14 of the pointing finger. Alternatively or in additionto determining the fingernail 26 of the fingertip, the method may detectthe index finger knuckle 28 on a left portion of the hand if the user 16points with the right hand, and on a right portion of the hand if theuser 16 points with the left hand. Alternatively, the system may beconfigured to detect a signature of a group of knuckles, such as theknuckles on the third, fourth and fifth fingers. The system might alsodetect the folded thumb, illustrated in FIG. 15, which is characteristicof a pointing hand. This may be useful because, as illustrated in FIG.15, a tip of a pointing figure is less distinct in its recognizablefeatures from other more distinct portions of the hand. However, oncethe more distinct portions of the hand are located, a location of thefingertip may be extrapolated.

More specifically, extrapolation can occur through measurement orestimation of the width, height, and/or location of the hand withrespect to the user's face. This data may help estimate the particularanatomy and dimension of the user's hand, and thus aid in predicting thelocation of the fingertip 14, if it is not readily apparent from thecaptured images. Further, known relationships between the spacing of thefingers may help improve the estimation. Jewelry, such as one or morerings may be used as reference points to determine the location of thefingertip 14. Reference data saved in memory 13 for a particular user 16or average anatomical data may assist in improving the estimation. Anyof the reference data discussed above may be used in an algorithmassociated with identifying a pointing location. The reference point mayinclude, for example, a central location of the hand, jewelry, any ofthe fingernails (including the fingernail 26 on the fingertip 14), andone or more knuckles, such as index finger knuckle 28. Once a referencepoint has been identified, the algorithm may calculate an offsetdistance in x- and y-coordinates or in x-, y-, and z-coordinates as anestimate location of the fingertip 14 with respect to the referencepoint. The method may identify and track one or more of the referencepoints and apply the corresponding offset associated with the referencepoint to estimate the location of the fingertip 14.

Other embodiments of the invention may include a device for calibratinga touch-free gesture recognition system, the device may include at leastone processor, configured to: enable presentation of a prompt to a userto make a touch-free gesture at at least one prompt location on adisplay; receive information from a camera indicating a correspondinggesturing location on the display, wherein the gesturing locationdiffers from the at least one prompt location as a result of anon-alignment between the gesturing location on the display and anintended gesturing location of the user; and calibrate the gesturerecognition system to account for the non-alignment.

In addition, in the device for calibrating a touch-free gesturerecognition system the at least one processor may be further configuredto: use the touch-free gesture for at least one operation in addition tothe calibrating; enable the prompt to occur during use of an applicationwithout identifying to the user that calibration is occurring; increasegesture location accuracy periodically during use of an application byperiodically recalibrating the gesture recognition system to account fordifferences between detected locations of gesturing on a display duringuse and locations of icons on the display that the user is suspected ofattempting to select; predict a user's intent to activate a specificicon based on proximity of the gesturing location on the display to thespecific icon, and to calibrate based on the prediction; permit periodicrecalibrating during normal use of the application without uniquecalibration steps required by the user; delay the calibration untilafter the user performs a subsequent action following the touch-freegesture at the at least one prompt location; increase the gesturelocation accuracy by recalibrating during an appliance unlockingprocess; determine, based on a layout of icons on the display, whether auser's intent is to activate a specific icon when there is a differencebetween icon location and the gesturing location; prompt the user tounlock the associated appliance after detection of a predefined usermovement relative to the display; save information related to aplurality of pointing locations and relative locations of iconsassociated with each of the plurality of pointing locations, andcalibrate the gesture recognition system using the saved information;save information related to the gesturing location and a relativelocation of an icon associated with the gesturing location, and/orcalibrate the gesture recognition system on the saved information; andcalibrate, during an unlocking process, to account for uniquecharacteristics of the user's vision.

In addition, in the device for calibrating a touch-free gesturerecognition system the prompt may include at least one icon for cuingthe user to unlock an associated appliance or at least one icon, whereinthe touch-free gesture is a pointing gesture, and wherein the at leastone processor is further configured to cue the user to point at the atleast one icon. The at least one prompt location on the display mayinclude a first known location and a second known location, and whereincalibrating is based on information associated with prompting the userto gesture from the first known location to the second known location.The predefined user movement may include at least a body portion of theuser moving out of a viewing area of the camera.

The gesturing location may be a pointing location, and wherein the atleast one processor may be further configured to: initially calibratethe gesture recognition system based on the pointing location and alocation of at least a first icon; after initial calibration, enable theuser to control an application using gesturing; during control of theapplication, gather information relating to the gesturing relative to atleast a second icon on the display; and/or recalibrate during control ofthe application when the user attempts to point at the at least secondicon.

The non-alignment may be a result of one eye of the user being dominantover another eye of the user, and wherein the at least one processor maybe further configured to: calibrate the gesture recognition system toaccount for the one eye of the user being dominant over the other eye ofthe user; determine an offset between the gesturing location and the atleast one prompt location; and/or determine an adjustment to apply tosubsequent gesturing location determinations based on the offset.

Other embodiments of the invention may include a touch-free gesturerecognition system, including at least one processor, configured to:enable presentation of a prompt to a user to make a touch-free pointinggesture at at least one prompt location on a display; receive imageinformation from an image sensor corresponding to the touch-freepointing gesture; determine a fingertip location associated with thetouch-free pointing gesture, the fingertip location being determined, atleast in part, based on an anatomical feature of the user other than atip of the user's finger; and determine a pointing location on thedisplay associated with the touch-free gesture based, at least in part,on the determined fingertip location.

The touch-free gesture recognition system may also include at least onememory device for storing a plurality of algorithms for determiningpointing locations based on differing hand orientations, and wherein theat least one processor is further configured to select a preferredalgorithm from the plurality of algorithms based, at least in part, on ahand orientation in the image information, and to determine the pointinglocation based, at least in part, on the preferred algorithm.

The touch-free gesture recognition system may also include at least onememory device for storing a plurality of algorithms for determiningpointing locations based on whether a right hand or a left hand isdetected in the image information, and wherein the at least oneprocessor is further configured to: select a preferred algorithm fromthe plurality of algorithms based, at least in part, on whether theright hand is detected or the left hand is detected, and to determinethe pointing location based, at least in part, on the preferredalgorithm; and/or determine whether the hand of the user hand is a lefthand or a right band based, at least in part, on an offset between theanatomical feature of the user and at least a portion of the user'shand.

In the touch-free gesture recognition system, the anatomical feature:may include a fingernail; may be at least one of a width, length, size,or location of a hand of the user; and/or may be a second fingertipother than a fingertip used in the pointing gesture, and wherein the atleast one processor is further configured to determine the fingertiplocation based, at least in part, on the fingertip used for pointing andthe second fingertip.

In the touch-free gesture recognition system, the at least one processormay be further configured, in an absence of locating the fingertip inthe image information, to: determine the fingertip location based, atleast in part, on a predefined area of a hand of the user other than afingertip; and/or determine the fingertip location based, at least inpart, on an estimated length of a finger used for pointing.

In the touch-free gesture recognition system, the at least one processormay be further configured to: determine whether a hand in the imageinformation is a left hand or a right hand, and determine the fingertiplocation as a location associated with a top-left area of the hand if aright hand is determined or a top-right area of the hand if a left handis determined; locate a hand by applying to the image information afilter mask associated with a size of the hand; detect two fingers of ahand of the user in the image information, and determine the fingertiplocation as a location between the two fingers; and/or detect aplurality of fingers of a hand of the user in the image information,determine the fingertip location based, at least in part, on theplurality of fingers, wherein each of the plurality of fingers isassociated with a differing weight for determining the fingertiplocation.

In the touch-free gesture recognition system, the at least one processormay be further configured, after determining the fingertip location at afirst time in an absence of locating the fingertip in the imageinformation, to subsequently track the fingertip using non-fingertipinformation.

In the touch-free gesture recognition system the at least one processoris further configured, after locating the fingertip location at a firsttime, but failing to locate the fingertip location at a second time, tosubsequently track the fingertip using the fingertip location at thefirst time and non-fingertip information.

Other embodiments of the invention may include a device for providingvisual feedback in a touch-free gesture recognition system, the devicecomprising: at least one processor configured to: enable presentation ofan image on a display; receive information from a camera, theinformation being indicative of a touch-free gesture performed by auser; enable presentation of a gesturing indicator on the display,wherein the gesturing indicator is indicative of a location of theuser's touch-free gesture relative to the display; present the gesturingindicator in a first visual appearance on the display when the user'stouch-free gesture is a first distance from the display; and present thegesturing indicator in a second visual appearance on the display whenthe user's touch-free gesture is a second distance from the display,different from the first distance.

In the touch-free gesture recognition system the gesturing indicator maybe a pointing indicator. The information received from the camera may bereflective of at least a portion of the user's hand. The image on thedisplay includes at least one icon.

In the touch-free gesture recognition system the at least one processormay be further configured to: decrease a size of the gesturing indicatoras the distance of the user's touch-free gesture from the displaydecreases; increase a size of the gesturing indicator as the distance ofthe user's touch-free gesture from the display increases; detect acorrective movement associated with the user's touch-free gesture, andto determine a user's intent to select the icon based on the correctivemovement; present the gesture indicator only after a portion of thegesture has been detected and/or when a hand of the user is detected ina predetermined area in the field of the image sensor; present thegesture indicator only after a detected location of the gesture motionrelative to the display falls within a predetermined threshold; increasea size of the gesturing indicator as the distance of the user'stouch-free gesture from the display decreases; and/or decrease a size ofthe gesturing indicator as the distance of the user's touch-free gesturefrom the display increases.

In the touch-free gesture recognition system the display may be avirtual 3D display, and wherein the at least one processor may befurther configured to: cause a distance between the gesture indicatorand the display to change as a distance between the user's hand and thedisplay changes; cause the virtual 3D indicator to appear as ifemanating from at least a portion of the user's hand; decrease the sizeof the virtual 3D indicator as the distance of the user's touch-freegesture from the display decreases; and/or display the gesturingindicator in two forms, a first for prior to a selection of informationon the display, and a second form after selection of information on adisplay.

It will also be understood that the system 2 according to the presentlydisclosed subject matter may include a suitably programmed computer.Likewise, the presently disclosed subject matter contemplates a computerprogram being readable by a computer for executing the method of thepresently disclosed subject matter. The presently disclosed subjectmatter further contemplates a machine-readable memory tangibly embodyinga program of instructions executable by the machine for executing themethod of the presently disclosed subject matter.

It is to be understood that the presently disclosed subject matter isnot limited in its application to the details set forth in thedescription contained herein or illustrated in the drawings. Thepresently disclosed subject matter is capable of other embodiments andof being practiced and carried out in various ways. Hence, it is to beunderstood that the phraseology and terminology employed herein are forthe purpose of description and should not be regarded as limiting. Assuch, those skilled in the art will appreciate that the conception uponwhich this disclosure is based may readily be utilized as a basis fordesigning other structures, methods, and systems for carrying out theseveral purposes of the present presently disclosed subject matter.

Certain features which, for clarity, are described in this specificationin the context of separate embodiments, may also be provided incombination in a single embodiment. Conversely, various features which,for brevity, are described in the context of a single embodiment, mayalso be provided in multiple embodiments separately or in any suitablesub combination. Moreover, although features may be described above asacting in certain combinations and even initially claimed as such, oneor more features from a claimed combination can in some cases be excisedfrom the combination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Particular embodiments have been described. Other embodiments are withinthe scope of the following claims.

1. A touch-free gesture recognition system, comprising: at least oneprocessor, configured to: enable presentation of first displayinformation to a user to prompt a first touch-free gesture at at least afirst location on a display; receive first gesture information from atleast one image sensor corresponding to a first gesturing location onthe display correlated to a first touch-free gesture by the user,wherein the first gesturing location differs from a location of thefirst display information at least in part as a result of one eye of theuser being dominant over another eye of the user; determine a firstoffset associated with the location of the first display information andthe first gesturing location; enable presentation of second informationto prompt the user to make a subsequent touch-free gesture at at least asecond location on the display; receive subsequent gesture informationfrom the at least one image sensor corresponding to a subsequenttouch-free gesture by the user; and use the first offset to determine alocation on the display affected by the subsequent touch-free gesture.2. The system of claim 1, wherein the first touch-free gesture and thesubsequent touch-free gesture include finger pointing and wherein the atleast one processor is further configured to use the first gestureinformation and the subsequent gesture information from the at least oneimage sensor to determine pointing locations on the display.
 3. Thesystem of claim 1, wherein the at least one processor is furtherconfigured to send an output to a destination reflective of a recognizedgesture associated with the subsequent touch-free gesture.
 4. The systemof claim 1, wherein the subsequent gesturing location differs from thelocation of the second information on the display at least in part asthe result of one eye of the user being dominant over another eye of theuser.
 5. The system of claim 1, wherein the at least one processor isfurther configured to determine the first offset by identifying at leastone of a first candidate pointing area generated using at least one of afirst ray associated with a first eye of the user and a second candidatepointing area generated using a second ray associated with a second eyeof the user.
 6. The system of claim 1, wherein the informationassociated with the first offs corresponds to a location on a face,other than a pupil, to serve as a starting point for a line of sightcalculation, and wherein the at least one processor is furtherconfigured to use the non-pupil location on the face to determine anintended location of the subsequent touch-free gesture.
 7. The system ofclaim 1, wherein the at least one processor is configured to use theinformation associated with the first offset to determine an intendedlocation on the display of the subsequent touch-free gesture.
 8. Thesystem of claim 1, wherein the at least one processor is furtherconfigured to determine the first gesturing location by selecting atleast one of a first point within a first region on the displayassociated with a first line of sight passing from a location associatedwith the first eye, through a fingertip, and onto the display, and asecond point within a second region on the display associated with asecond line of sight passing from a second location associated with thesecond eye, through the fingertip, and onto the display.
 9. The systemof claim 8, wherein the at least one processor is further configured todetermine the first gesturing location by determining at least a thirdpoint in the general vicinity of the first point and the second point.10. The system of claim 1, wherein the at least one processor is furtherconfigured to determine differing offsets associated with the eyedominance for differing regions of the display.
 11. The system of claim1, wherein the at least one processor is further configured to determinea location of pointing by using multiple candidate pointing area regionsat a plurality of differing time intervals.
 12. The system of claim 1,wherein the at least one processor is further configured to store uniqueinformation associated with the first offset for each of a plurality ofusers.
 13. The system of claim 12, wherein the at least one processor isfurther configured to store, as the stored unique information,information received from the at least one image sensor associated withan identity for each of the plurality of users.
 14. The system of claim12, wherein the at least one processor is further configured to store,as the stored unique information, information related to a non-pupillocation on a face for each of the plurality of users.
 15. The system ofclaim 1, wherein the information associated with the first offsetincludes an indication that the one eye of the user is dominant over theother eye of the user.
 16. The system of claim 1, wherein theinformation associated with the first offset includes an indication of adegree to which the one eye of the user is dominant over the other eyeof the user.
 17. A non-transitory computer-readable medium comprisinginstructions that, when executed by at least one processor, cause the atleast one processor to perform operations including: enablingpresentation of first display information to a user to prompt a firsttouch-free gesture at at least a first location on a display; receivingfirst gesture information from at least one image sensor correspondingto a first gesturing location on the display correlated to a firsttouch-free gesture by the user, wherein the first gesturing locationdiffers from a location of the first display information at least inpart as a result of one eye of the user being dominant over another eyeof the user; determining a first offset associated with the location ofthe first display information and the first gesturing location; enablingpresentation of second information to prompt the user to make asubsequent touch-free gesture at least a second location on the display;receive subsequent gesture information from the at least one imagesensor corresponding to a subsequent touch-free gesture by the user; andusing the first offset to determine location on the display affected bythe subsequent touch-free gesture.
 18. The non-transitorycomputer-readable medium of claim 17, wherein the subsequent gesturinglocation differs from the location of the second information on thedisplay at least in part as the result of information indicative of oneeye of the user being dominant over another eye of the user.
 19. Thenon-transitory computer-readable medium of claim 17, wherein theinstructions, when executed by the at least one processor, furthercauses the at least one processor to perform an additional operationincluding determining the first offset by identifying at least one of afirst candidate pointing area generated using at least one of a firstray associated with a first eye of the user and a second candidatepointing area generated using a second ray associated with a second eyeof the user.
 20. The non-transitory computer-readable medium of claim17, wherein the information associated with the first offset correspondsto a location on a face, other than a pupil, to serve as a startingpoint for a line of sight calculation, and wherein the instructions,when executed by the at least one processor, further causes the at leastone processor to perform an additional operation including using thenon-pupil location on the face to determine an intended location of thesubsequent touch-free gesture.